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Abatrack , 

The Feldt-Gilmer congeneric reliability coefficients make it possible 
to estimate the reliability of a test composed of parts of unequal, un- 
known length. These coefficients include Kristof 's coefficient and 
Cronb,ac#s alpha' coefficient as special .cases. In this paper the approxi- 
mate standard errors of the Feldt-Gilmer coefficients are derived via a 
method that uses the multivariate Taylor's expansion. Monte Carlo slaiti- ' 
lation is employed to corroborate the theoretical approximations for 

I 

eighc. hypothetical tests. It is shown that the Feldt-Gilmei: coefficients 
are appropriate with both congeneric and tau-equivalent parts. Their 
superiority over the Krxstof coefficient increases with the number of 

t 

parts (K > 4). Their standard errors also compare favorably to that of 
Cronbach's alpha when applied to tests with tau-equivalent parts. 



IntroduoHon 



• Measurement situations sometimes occur in which a multiple-part 
measure can be assumed to consist! of 'congeneric parts,. but not essentially 
tau-equivalent parts. Congeneric parts are' parallel in content and mea- 
sure the same attributes, but differ in length. Tau-equivalent parts are 
parallel in content and functionally equal in lehgth. More formally, 
parts j and h are congeneric if their true scores satisfy the relation 
T,; = aTjj + b . where a is a constant not necessarily equal to 1.0 and 
b^is a constant not necessarily equal to 0 . Parts j and h are tau- 
equivalent if a = 1.0 . For both types of parts error score variances 
and observed score variances may be heterogeneoui from part to part. 
For congeneric parts the ttue score variances are, in general, hetero- 
geneous. For tau-equivalent parts true, score variances are homogeneous. 

Examples of congeneric pa*ts are not difficult to find. In -reading 
tests the passages often vary in length and in tlje number of items based- 
on each passage. If these numbers of items vary appreciably, the passage 
scores will very likely be congeneric. The ratings of individual judges 
on a panel will- be congeneric if some judges concentrate their ratings 
in the midd> of the scale |tnd avoid extreme ratings, while others spread 
their ratings over the full range. In an essay test, if the several 
, questions differ in difficulty or are given different point values by 
the examiner, the' scores on the various questions might well be con- 
generic rather than essentially tau-equivalent. 



Coefficient alpha iCronbach, 4951] provides an , acceptable estimate 
of reliability If tbe part scores are essentially tau-equlvalent [Ndvlck ^ 
ano Lewis, 1967]. Horst's coefficient [1951] will serve for congeneric 
parts If the number of Items can be assumed to represent the effective 
or functional length of each part. In the case of ratings and essay 
tests, however^ the lengths of the parts cannot be Inferred from any ^ 
obvious feature ofi the parts. In such situations, one must use one 
or anqjther of the approaches develeped by Feldt 11975>, Krlstof [1974], 
or Gilmer and Feldt [1981]. 

The first purpose of this study was to develop approximate standard 
error formulas for the last two of these congeneric reliability coeffl- 
clents. The basic technique used for this purpose Is the "delta" method 
of Kendall and Stuart [1969, Volume I, Chapter 0.0]. This method makes 
It possible to approximate the standard error of certain' statistics (to 
order "N"^) even though the sampling distribution of the statistic is 
unknown. The second purpose was to verify the comparative values of 
these standard errors through a Monte Carlo simulation of the sampling 
distributions of the coefficients. The last purpose was to draw some 
conclusions regarding the choice of estimates* on the basis of their 
bias and standard error. 

1%^ CongeneHo Coeffid^nte 

Because the derivations of these coefficients have been p^v -*nted 
in detail elsewhere [Giliper, 1981], they will not be repeated here. 



However, the notational scheme and the structure of the congeneric model 
must' be^ made explicit. 

The congeneric model can be specified- as folldws for a test that 
consists of K parts: 

' X - + X2 + X3 + • . • + 

^ T = Tj^ + T2 + + • • • + Tj^ 
X - T + B 

X - + E, =*X.T + b, + E/ (j - 1. . . . , K) 

|. j j j 3 J J. 

Z\. - 1.0, X.> 0 

Zh." = 0 - . 

J • 

In this representation X, T, and E are the observed, true, and error 
scores for the total test, respectively; X^ , , and E^ (j = 1, . . • K) 
are part-test observed, true, and error scores. The are constants 
which tepresent the proportions of the total test true score that are con- 
tributed by the various parts. They nay be viewed as the "proportionate 
functional lengths" of the parts. The b^ are constaats which make allow- 
ance for the difl'erences in mean score on the parts, beyond differences ' 
arising from variation in part-test length. The error scores, E^ , are 
assumed to be mutually independent and independent of true scores. The 
linear relationship among the part-test true scores implies that they are 
perfectly correlated with the true scores on the full test. 



'r " 
6 




AddjLtlonal parameters of importance are as follows: 

« covarlance of perts 1 and J ^ 
■ variance of part J 

» total test true score variance, and v 

r 

otal test observed score variance. 

Given the Independence of , Ej , and Tj the observed score 
variances and covarlance^ ar^ as follows: 

°ii ' ^J*^T (J - 1, .... kJ, 

The total of the covarlatices In row 1 of the part-test covarlance matrix 

18 

In the part- test covarlance matrix for any e^camlnee sample, let the 
row* with the largeilt sum of covarlances be designated as row i . Desig- 
nate by Cj the quotient obtained by dividing the sum of covarlances in 



row J by the sum of covarlances In row I . Let Y * Thep, 

• •• ••• o*** 



Consider the pair of functions: * • ^ ^ 

' f^(^) - I - 1 + J -25-? - y.25-CjY - 0 , and (la) 

fjj(Y) - I - 1.- ^ ✓ .25-Y - 0 . (lb) 

One or the other of these functions must be solved for Y , the choice 
being dictated by the algebraic sign of fj(,25). The solution for Y is • 
then substituted into the following fennula to obtain an estljnate of the 
total test true score variance; 

Using the variance ratio definition of reliability ,. we then obtain 
the first congeneric coefficient: 

For the second congeneric coefficient, designated r^^ , we again 
identify the row with the largest sum of covar lances as row I . We 
then obtain the following quotients: ^ 



ft 



• • • 



The second congeneric. coefficient is then defined as 

(ZD,)^ ^2 r§ - 
r - J • — U • , (A) 

The^ last congeneric coefficient considered 'In this paper is that 
derived by Kifistof [1974] for a three-part test. To , apply this estimate 
of reliability to a Tonger test, one must combine parts and reduce the 
instrument to three parts. The formula for Kristof's coefficient is 

, (5) 



(^12^13 ^12^23 •■'W23>' 



'^k" (^12>(^13>(^23>^x 
,If a test consists of only three parts, sample estimates r^^^ r^^ , and 



r-. will be identical • # « 

Approximate Sampling Variances ' 

A method by which the sampling variance of a statistic can be deter- 
mined when the sampling distribution is unknown is presented by Kendall 
Wid Stuart 11969, Volume I, Chapter 10^]. This method, called th6 delta 
method, is applicable when the statistic of interest is a function of 
"simpler" statistics, the sample values of which are always greater than 
zero* With sufficiently li-rge samples, this coa4ition can b^ met in the 
present context* 

The delta method uses the initial terms in the Taylor expansion of 
a function of several variables. Let X - (Xj^ , • t Xp)' and a • 
(a- , a^- , . . . , a . Then the terms through the firdt derivative are: 

*^ p 



f(X) - f(a) + SCV^i^^iCa) • 



(6) 



Subtracting f(a) ^rom both si^, we 'get 

^ P , . 

• f(X)-f (a) - |(X^-a^)f^(a) . 

•and squaring, both sides yields the relation » 

, PP 

[f(X)-f(a)]^ - ZZ(X^-a^)(Xj-a^)f'(a)'fj(a) . 
If a = £(X) and^[f ^X)l - f(a) , then taking the expected value of ^ 

t 

both sides of ( 6 ) gives an expression for the variance of f(X) : 
VaT[f(X)l - e[mXi-a^)(Xj-aj)q(a)f'(a)] ' 

- ZZ[Cov(X ,X )f:(a)f'(a)l . 
• ij \ 
Define £(X) - e and f(e) - g . We then have 



Var[f(X)l - II 
ij 



3-g . »38 . Cov(X ,X.) 
38^ 36^ 



(7) 



This last Equation is, the same as equation 10.12 of .Kendall and 

* 

Stuart [1969]. ^ \ 

; In the context of this paper tn« X^ represent the sample variances 
and covariances of - the part tests, i.e., t;^e elements In the sample 
varlance-covariance matrix; the 6^ are the population variances and 
covariances of the part tests. The f(X), in our case, is any oile of the 
reliability estimates discussed previously. Using r as a particular 
reliability estimate and p as its population counterpart, we. obtain 
from (7): ' ^ 



Var(r) 



3p . Coig<X^ 



36 



(8) 



10 



It is necessary^ then, to determine Cov(3Cj^ ,X* ) ,»the covarianqe of 

two covariances. , In particular, we need Cov(S . ,S ), where ,m,n, - 

1, •••5 K| K • the i^uniber of \part tests and S represents th^ sample 

estimate of a^. • ' • • ' ' • ^ 

Tfce determination of t€ov(S. . ,*S_) is not straight forward 
^ \ ij mn 

Kendall and Stuart develop the dovariance of two^ covariances with the 

1 \ ^ ■ ^ 

use of Fi-sher's^TStatistics [^i^er, 1928, in Xendall & Stuart, i969. 

Volume I]. The eod, result of this extended derivation is as follows :^ 

^ o \ ^im^jn + ^in^jm . . 
^fJ, ^ N-l . 

The variance obtained from (8) is" not exact. It is correct ^to the 

drder of n"^ . ' Therefore, the approximation and the true value of the 

\ ••23'' 
sampling variance may differ by ap amount that involves .N , N , 

In the denominator. With N relatively large, *ay N V 100, the difference 
would probably not be of great Importance, ' ^ . 



TheoreHoal Variqjice ^p2 ' " 

As in the foregoing section, we presume that the divisor for Gj is 
the maximum sian.of row covariances. Withbut loss ofc generality we shall 
assume tl^s maximum occurs for row 1. \ Since the delta method is applied 
to a function- of parameters, ttie following relationships are defined 
among the parameters in the poptilation vafstance-covar lance matrix: 



« t 



OLi. 



+ a 



2K 



- la 



ij 

2j 



'1 a 



- 1.0 , 



p * 



!2 . 

01,. 

J. 
a. 



'3 a 



1^ 



a. 



symbol Yj is used here, rather than \ to emphasize that 
are ratios of parameters. ' The papulation equation corresponding to 
eqiiatrons (la) and (lb) is 

, : h.<0,j)-g(aij.Y)-f-i±y^- ' 



these 



(10) 



/.25-Yy. = 



0 . 



The true score variance, , . is a^/Y . and the reliability is 

\ 
\ 

2 



^Fl 



2 2 



The variance of the sample estljwrte of p^^ is then given by 

KV KV 

Var(r, 



/ 



^Fi> -£ 

b-1 



^Ppi . !!f1 . Cov(3e .X^) 
^«d 



10 



In this expression KV ^ K(K + l)/2, the lotal nmnber of distinct elements 
in the variance-covariance matrix, the parameters 9^^ through 9^^^ ar^ the 
distinct part-test covariances (KC - K(K-l)/2); 9^^^^^^ through 9^^ are the 
part-test Variances. The statistics through are the sample esti- 
mates corresponding to through 9^^ . For .example, if\K » 4, the fjllow- 
ing vould be defin^: \ 



\ KV 


m 


10, 


KC 


m 


6, 


«^ 


m 




^1 






«2 


m 


*^13 




m 




«3. 




*^14 


h 




hi. 






^^23 






^23 






°2U. 


h 




^24 


. \ 


m 


*^34 


h 


m 


«3A 






*^11 


h 


m 


hi 




m 




h 


ss 


^22 




m 


^33 




m 


^33 


• ho' 




. *10 


m 


hi.. 



Note that the first K-1 terms ate the covariances from row L, the covari- 
ances in the sum o, . Note also that the elements in the lower triangle 
of the population variance-covariance matrix have not been included expli- 
citly.^ This means tha^: when we refer to 9^ (b « 1, • • • KV) , that is, when 
we refer to one of the , we imply that 1*1 J . This fact, that 1 < j, 
will nymtfir again, in this derivation. 



11 



The covariance of S^^ with S^^^^ is, froas (9). 



Im jn in jm 
^ N-1 



where N is sample size. Once ^Q^l^^^h * ^» " * • found, the 

variance of r can be approximately determined from expression (n) for 
Fl 

any given population matrix and specified N . 

The first step in this process is to apply the chain rule to 



Fl' 



b 



39. 



31 



2 2 

Thf§ result indicates that we need ^^^^^^^b ^^x^^^b * 
' The first partial derivative is straightforward : 



39. 



39, 



I2 if b - 1, KC 

[l if b - KC + 1, KV . 

2 

The partial derivative of 3a^/39^ is not as direct:. 

2 



3a^ 
39T 



30, 
39, 



, if b - 1, KC, 



0 , if b - KC + 1, KV . 



(12) 



(13) 



14 



12 



We have then\ 



^ ^^b ^ , If b - 1, KC 



8p 



Fl 



a 



, If b - KC + 1, KV. 



We need to determine 3a^/3e^ for b = 1, KC: 



(14) 



2 1^ 

!!t li _^\_^J!b , b - 1, KC. 

We now need ia^/iQ^ and 3Y/3e^j . The first of these is 



(15) 



3a J 
36^ 



3( ra ) 
36, 



(16) 



'b b 

' 1 if b - 1 , • • • , K-1 

[0 if b - K, KC. ^ 

The determination of 3Y/3e^j involves implicit differentiation because 
Y fs given Implicitly in (lO)- We have from (10): 
h(aj^j) - g(a^j.Y) - 0. 



ERIC . 



15 



\ 



13 



Therefore, 



ERIC 



3a, „ ^ 3a,, ^„ 3a. 



dh dt_ . 3fi_ .\ 12 _3£_ . • -J± + '•' ' 

di^-de^ 3a^2 '"13 ''b '^^14 ^\ 

which simplifies to 

dh_„ iS_ + M • 0, b-1, •••,KC. 

de^ 36^ 3Y ^ • 

SolYins for 3 Y/3e, , as r equired, we obtain; 



39 = 3& 
° 3Y 



y2 . 3& 
3Y 



where ' 



(17)\ 



Substituting the right sides of (16) and (17) into (15) we obtain: 

a, Y.I. ^. a, 

_I _^ 3Y 
36^ -36^-^2 



(18) 



I - 1 if b - 1 , • • • , K-1 
I - 0 if b - ,K, KC. 

Again, we ne4d two partial derivatives: 3g/Se^ , for b - 1^ * * * , KC, 
and 3g/3Y . 



16 



14 



Ck>n8lder first 3g/3Y . From (10) we caa obtain: 



where 



3Y 



2 y .25-Yy. 



Thus- we have 



If-? i + E 



The minus sign is used if X^^ > .5 , otherwise the plus sign is used. 
Now consider the numerator of (17) From (lOO we can obtain 



where 



'\ m 



3( y .25-YYjj) _y 3y 



So we have 



"b Ml 2 y .25-YY,. "b 



17 



15 



What now remains to be determined is ^yy./^\ • Applying the chain rule 
we obtain: 

!lk °1 " . b ■ 1. KC. (21) 

36, " 39. " J . 

b b . " 

And once more we need two partial derivatives: 30^^/36^ and 30^/36^^ , 
b » 1, KC. • 

\ 

Consider 3ci^/38j^ first: 

!\ - ^"^^cicU ^LlfM^^/ ^ 

\ 

lifk-iork-j, 

0 if k / i and k ^ i. - , 



Mow ^onsider 30^^/36^ : 



3y. 3Yi, a,P-a,Q 

Jk._UE L_A- , i < j. b - 1, KC. 



-± i2 1| IK^ . i < j. (23) 

^^b ^^ij 



1 if i - 1 
0 if i ^ 1. 

Combining ( 22 ) and ( 23 ) into ( 21 ) we obtain 



36. 30.. 2 
b ij. 



(24) 




18 



16 



where 




In 8\iimnary» then, we have 

D 



3p 



Fl 



-< 



, b - 1. KC 



-a 
a 



, if b - KC + 1, KV. 



36. V2 . M 

3Y 



(25) 



(26) 



where 



I - ' 



1 if b - 1, K-1 
0 if b - iCi KC 



1 . _ ^ 

- 2 J .25-Y kjtl 2 y.25-YYj 

^®b 2 /l5^ 



(27) 



(28) 



19 



RJC 



-1 



17 



Where 



, 1 < J, b - 1, , KC 



lifk-lort-J 



0 otherwise 



1 if 1 - 1 
0 Otherwise 



Finally, by„ substituting (2^ ) into (25 ) we obtain 



• 






36, 1 


^ , 3Y 








2 




r » 


4 


9 



(29) 



-2o; 

T 

if b - 1, .... KC 

if b - KC + 1, .... KV 



(30) 



where 



I -< 



1 if b - 1, 
0 If b - K, 



K*l 
KC, 



and 3g/3y and 3g/39^ are determined by expressions;(27 (28 ) and (29 ). 

Clearly, the dietermlnation of th6 theoretical variance of r^ involves 

/ 

considerable computation. In the present study computer progl;ams were 

rj 

developed to evaluate t;He multitude of terms arising for Illustrative 
joatrlces presented In a later section. 



SO 
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' 'J%eap0tioal Vaxdanoe of r^g 

The population counterparts to the oa.page 5 will henceforth be 
represented by • In this notation. 



where is defined as before, nanely.the sum of the population covari- 
ances in row j. t- ' •* 

• - V 

The population reliability, analogous to equation ( 4 ) , is 



(SB, 



- sey- 



By equation ( 8 ) the variance of r_- is 



KV KV . 
Var(r^2>-I Z 



D Q \ 



b-1 d«l 

where the 6^ and are ar^ defined as beforVi^^g^t 



- ,c c ^ ^li.^n '^in'^lm . 

What we now need to find is 3pp2/'^b " 1» * * ' » ^® ^"^^ 



t 2x2 



, b- 1, KV. 



21 

\ 



19 



Equations (12) and (13) hold in this situation: 

* / ° 



2 if b - 1 , • • • , KC 

•b - KC + 1, ••• , KV, and 



1^ 1 if 



'4 



Equation (14) also holds: 



0 , if b « KC + l,^--, KV. 



2 ^*^T 2 

s- , if b - i; • • • , KC 



3p 



F2 



< 



J. 

4 



, if.b - KC + 1, KC. 
We need to find 0^39^, . for b - 1. . KC, where^e^crue s^o^ 



2, 



^rfance mayHoe written 



IZ a 



1 - 



1 



20 



Then we have 



3 1 - 



"I 



3a^ ^ (Eg^) j b . i^J / b 



(31) 



Clearly, we need the two partial derivatives: 



3 1 - 



"1 



^==1-1= i~aad — — — , f or b. - 1 , • • • , KC. 



Consider the «£lr8t : 



3% " » 36^- 



(32) 



Consider the second: 



E6j ^\ 

<re.)7 \ (?e.)^ 

J-l - > \ , J ' , b«l, ....KC. (33) 



36, 36, 
b b 



4 



21 



m * 



Then 



96, 



(34) 



So we nepd 3(ZBj)^/30^j and 3EB^/3e^ , for b - 1, KC. We then obtain: 



2U 



2 



(35) 



and 



2 2 



30» 



96. 



j 38, 



2U 



j 30. - 



(36) 



Now we need only 3Bj/8e^ . We obtain 




3(a -6,.) 



3a 



ntn 



(37) 



/ 



22 



r 



We need then 5(aj-<^jt)/5% ^^°fc'*^jfc^^^^iBn * "® m < h, since 
b • 1, • •• f KC. So ' 

3(a.-o. J f 1 if (ft # n and J - m) or (4 ^ m and J - n) , (38) 
^^jm I 0. otherwise, 

3(o.-o.,) I 1 if (a - n and J m) or - m and J n) , (39) 
Olothervlse. 

Substituting these results into ( 37) we obtain 

J- - ^ J ■ , (40) 



where 



1 if (ft n and J - m) or (ft ^ m and J » n) 
0 otherwise ^ « 



J - 



1 if (ft" n and J m) or (ft - m and J n} 
0 otherwise 

Substituting (34)', ( 35) "and ( 36) into ( 33) we obtain 



2 2 "'l 



0 



(41) 



d 25 

ERIC 



.-•/ 



23 



Summarizing, we have 

I 



3p 



F2 



36, 



-o 

a 



where 



2H - EZ iJ ^ ^ 



KC 



H 



, if b - KC + i. 



(A2) 



1 



and aH/3e. Is determined from C40 ) and ( 41) above. 

' b 

As in the 'case ^f the standard error of r^^^, the standard ?tror for 

r involves a large number of terms for even a relatively small value of 

F2 " 
K. For the illustrative matrices presented later, the evaliiation wa^^ 

accomplished via a computer program written for this purpose. 



fheovetiaal Variance^ 

To apply the delta method to 



p <^12\3^ ^12^23^^23^^ 
- ^ ' 



equation ( 8 ) must be evaluated for the following six parameters and 



their associated sample estimfited; 

i 1 



26 



24 







°12 


■ h 


m 


hi 




m 






m 








^23 


h 


m 


S23 






°11 


h 


m 












m 


$22 


^.6 




"33 


h 




S33 



We again obtain 



30" 
36. 



a« -rr — — 20„ 

X 36^ ^ T 



, if b - 1,2.3 



-a 
a 



, if b - A, 5, 6, 



(43) 



Tliis result indicates we must find fov^h 1,2,3. ^ It will be 

2 

easier to follow th^ subscripts if is expressed as a function of the 



'i • 



(6^62 ^ 6^63 ^ 6263)' 

e.e.e, - 



25 



Letting A - i^^^z -*' ®1®3 ®2®3^ ^ ' ^1^3* "® ' 



2 AT 



2 B ^ - 
3a; * 36. ^ 36. 
T b b 



36, 



B 



OAR ,2 3B 

2^36"^ 367 

D D 

B^ ' 



2BA(Z6^-ejj) - -A- |- 



i 



2A 



A[2(Z6^6^-6^)-Al 

Be, 



Now, 



b 1 



3a; . 



ft 

28 



26 



Then we have 



bo; 



aie.-2A 
T b 

6? 



., if b - 1,2,3. 



Substituting the foregoing result into ( 43 ) we finally obtain 

1 



3p 



K 





r 2 "1 




2 


af,e-^-2A 






L 


T 




A 











b - 1,2,3 



(44) 



a 



b - 4,5,6 



where A - e^B^ "^ ^1^3 ®2®3' ^i "® defined on pages 24-25. In 

evaluating the doubl.e suanation ( 8 ) for Var (r^^) , we again use^ 

^ ^ , !i5^inl!in^. 

We havs now derived formulas for the approximate sampling variance 

of r„ . r«« , and r„ . These expressions involve a large number of 
Fl ¥2 K 

terms, since the Index of each of the double summations in equation ( 8 ) 
runs from 1 through K(K+l)/2 , or K^(K+l)^/4 terms in all. Evaluation of 
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the sUMfition requires the nunerlcal 'value of the elements of the popu- 
lation variance covariance natrlx, quantities Which are never known in 
practice. However, they can be evaluated for any postulated matrix. This 
haa been done for eight illustrative matrices presented in the next 
section. In the following section we sunnarise the results of a Monte 
Carlo study undertaken to corroborate the theoretical approximations. 

Jlluetrative Applioatione of the San^ling 
Variocnae DerivoHona 

The foregoing formulas for the squared standard errors were applied 
to eight hypbthetickl measures with the variance covariance matrices 
presented in Tables f and 2. The purpose of these applications wai to 
gain some insight. into the comparative stability of the three reliability 
coefficients j The four matrices in Table 1 presume four-part tests and 
those in Tablle 2 eight-part tests. Within e«ch of these values of K 
two of the hypothetical measures have a population reliability (p) of .6 
and two otherls taive'a reliabUity of .833 . Within each pair of matrices 
associated wi^L a given K and p . one matrix represents tau-equlvalent ^ 
parts and the' other congeneric parts. (Ta^-equivalent parts exhibit 
homogeneous liter-pert covarlances; congeneric parts do not, except in 
the special cAse of tau-equivalence.) The tau-equlvalent partd for these 
hypothetical ineasures are not parallel in the classical sense, since the 
variances of the parts are unequal—a phenomenon, which Implies unequal 
errdr variance. 



_ TABLE 1 ' 
Population Covarlance Matrices: K " 4 



- .833 



- .6 



Essential Tau-Equlvalent 



Congeneric 



























21.375 


9.375 


9.375 


9.375 




38 


18 


19 


6 




■1 


.4~ 


9.375 


18.375 


9.375 


9.375 




18 


21.5 


9 


4.5 


h 


■1 


.3 


9.375 


9.375 


15.375 


9.375 












^9.375 


9.375 


9.375 


12.375 




12 


9 


11 


3 






.2 




<> 






1 


4.5 


3 


4.5 


\ 


■1- 




"48.375 


9.375 


9.375 


9.375 




'64 


18 


12 


6 


\ 


■1 


.4 


9.375 


39.375 


9.375 


9.^375 




13 


43.5 


9 


4.5 




■1 


.3 


9.375 


9.375 


2'3.375 


9.375 










9.375 

km 


9.375 


0.375 " 


20.375 




12 


9 


26 


3 




■1 


.2 










_ 6 


4.5 


3 


11.5 


^4 


■1 


•.1_ 
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Population Covarlance Matrices: K ^ 8 



Essential Tau-Equlvalent 



10.3A375 
6.3A375 
6.34375 

7.34375 2.34375 • 
5.34375 
2.34375 5.34375 ^ 
4.34375 

3.34375 , • 



14 


6 


4.5 


4.5 


3 


3 


1.5 


1.5 


^1 




6 


10 . 


4.5 


4.5 


3 


3 


-1.5 . 


1.5 


^2 




4.5 


4.5 


7.375 


3.375 


2.25 


2.25 


1.125 ' 


1.125 


^3 


m 


4.S 


4.5 


3.375 


8.375 


2.25 


2.25 


1.125 


1.125 


\ 




3 


3 


2.25 


2.25 


4.5 


1.5 


.75 


' .75 




m 


3 


3 . 


2.25 


2.25 


1.5 


4.5 


.75 


.75 






1.5 


1-5 


1.125 


1.125 


.75 


.75 


2.375 


.375 


h 




1.5 


1.5 


1.125 


1.125 


..75 


.75 


.375 


1.375 


h 





28.34375 

16.34375 .. 

15.34375 2.?4375 
18.34375 
12.34375 
2.34375 . 12.34375 

9.34375 
6.34375 
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6 


4.5 


4.5 


3 


3 


1.5 


1.5 


^1 


6 


20 


4.5 


4.5 


3 


3 


1.5 ' 


1.5 




4.5 


4.5 


16.375 


3.375 


2.25 


2.25 


1.125 


1.125 




4^5 


4.5 


3.375 


19.375 


2.25 


2.25 


1.125 


1.125 


h 


3 


3 


2.2S» 


2.25 


11.5 


1.5 


.75 


.75 




3 


3 


2.25 


2.25 


1.5 


11.5 


.75. 


'.75 


h 


1.5 


1.5 


1.125 


1.125 


.75 


.75 


7.375 


.375 




1.5 


1.5 


1.125 


1.125 


.75 


.75 


.375 


4.375 
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Vhen an llliMtratlve aatrlx confprvs to the tau-equivalent aodel, 
each part la of length 1/K by definition. When an illustrative matrix 
confdTMto the congeneric aodel, the largeat part is arbitrarily set 
equal to four tiaes the length of the soAllest p#rt. The lengths of the 
' parts, that is, the values of , are Indicated to the rigiht of each 
^ rov in the Batriz* The true ecore variance was arbitrarily set equal to - 
150 for all population satricea. 

The approximate atandard errors of the three coefficients computed 
for the eight illustrative natrices arcf presented la Table 3. When the 
part-tests are tau-equlvalent» the staiJidard error of coefficient alpha 

(r ) is- also included. It vas conputed via the delta method rather than 

a - 

from its tr*e sa^iling distribution » which is (l-p)F ^ • ia order to 

1» 2 

achieve greater comparability to the other standard errors. The divisor 
of all of the standard erroijs,/N^, has been omitted from the values 
In Table 3. Table 4 presents the standard errors evaluated for sample 

* 

sixes of 50, 100, and 200. , 

The saaple value theoretical standard error trill be 

effected, to a saall degree, by the rov sun chosen as the divisor in 
coi^ting Dj^ , • • • , . (The population paraneter p^j depend 
on this choice.) The results presented here are based on the use of the 
row one SOB as the divisor. Bowever, the varlancf s obtained by the use 
of other rowii arc very close to the results obtained here. 

The sas^le value of r^^ «id the variance of r^ are depmdent upon 
the ttaaerlcal values of the variances and covarlances for the three parts 
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TABLE 3 ! . 



4 ^ ^ 

Theoretical Standard Errors Tines y H-1 



Congeneric 



^P2 



' r. 



p - .6 

.6985 
.6993 
.7055 



.833 



.2937 
.2941 
.2961 



K - 8 



'Fl 
'f2 



.6149 

.6151' 

.7055 



.2571 
.2571 
.2939 



Essential Tau-Equlvalence 



'Fl 
'^F2 



.6512 
.6512 
.7765 
.6513 



.2744 
.2744 
.3288 
.2744 



K - 8 



^Fl 
'^F2 



.6052 
.6052 
.7765 
.6053 



.2536 
.2536 
.3288 
.2543 
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TABLE 4 

Theoretical Standard Errors for Three Sample Sizes 



Congeneric 







-P ■ 


.6 




P - 


.833 








50 


100 


200 


50 


100 ' 


200 




'fi 


.0997 


.0702 


.0495 


.0419 


.0295 


.0209 


- 4 


'k 


.0939 
.1007 


.0702 
.0709 


.04^5 
.0500 


.0420 
.0423 


.0295 
.0297 


.0208 
.0209 




'^Fl 


.0878 


.0618 


.0435 


.0367 


.0258 


.0182 


- 8 


'f2 


|0878 
.1007 


.0618 
.0709 


.0436 
.0500 


.0367 
.0419 


.0258 
.0295 


.0182 
.0208 


Essential Tau-Equlyalence 




'fi 

'f2 


.0930 


.0654 


.0461 


.0392 


.0275 


.0194 


- 4 


.0930 
.1109 


.065.4 
.0780 


.0461 
.0550 


.0392 
.0469 


.0275 
.0330 


.0194 
.0233 




r 
a 


.0930 


.0654 


.0462^ 


.0392 


.0275 


.0194 




'fi 


.0864 


.0608 


.0429 


.0362 


.0255 


.0180 


- 8 


'f2 


.0864 
.1109 


.0608 
.0780 


.0429 
.0550 


.0362 
.0469 


.0255 
.0330 


.0180 
.0233 




r 


.0865 


.0608 


.0429 


.0363 


.0256 


.0180 
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of the measure. Different values can occur from the different ways in 
lAich a four-part (or more) test is reduced to three parts. In general, 
the more nearly equal the parts, the smaller is the standard error. To 
achieve maximum part-teat homogeneity, the four-part tests were reduced 
to three by combining parts three and four. The eight-part tests were 
reduced to three by combining parts one with two, three with four, and 
five through eight. 

The results in Table 3 and 4 show that the standard errors of r^j^ 

and r each of which maintain the identity of all parts and all inter- 
FZ 

part covariances, decrease from K - 4 to K - 8. For example, under a 
congeneric test with p - .6 and N - 50, the standard error of r^^' is .0997 
with K - 4 and .0878 with K - 8. Thus, these coefficients show the same 

trend as r with increasing numbers of parts, 
a 

The tables also show that the standard error of r^, is larger than 
the others. Vith K - 4, the difference is small. With K - 8, however, 

the standard error of r„ is thirty per cent greater than that of r^^ and 

' ' 

r . It seems clear that the advantage of and r^^ over rj^ depends 
on the difference between the original K and K - 3. 

Another result of interest is the comparison of the standard errors 

of r , and r„» with each other and with that of . Even under tau- 
Fl F2 

equivalence, the standard errors of r^^ and r^^ c<»«P«e favorably to that 

of r . Moreover, the* more easily computed congeneric coefficient, r^^ » 

a ' 

is as stable as r_, . The latter estimate requires an iterative compu- 

* FX 

tational procedure which la! feasible by h^nd, but rather tedious. 
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• Memte Gxrlo Confirmation of the 

Theoretioal Standctrd Errors 

The population matrices in Tables l^d 2 were^used to generate 
aaaple matrices through a procedure developed by Odell and Feiveson [1966] 
Okd Browne [1968]. Two thousand sample matrices were generated under each 
of the twenty-four combinations of the following conditions: 
Number of part-tests: K « 4 and K « 8 

Population reliability: p - .833 and .6 
Model: tau-equlvalent and congeneric 

Sample siee: N » 50*, N - 100, N - 200 • 

All of the foregoing coefficients .were computed for each of 2000 
sample matrices under each configuration. In addition, the Mayekawa and 
Haebara [1980] least squares coefficient for congeneric tests, designated 
r , was coflq>uted for comparative purposes for each sample matrix* The 

liS V 

eavlrical ^andard errors are reported in Table 5. The differences be- 
tween the ilonte Carlo and theoretical starHard errors, and the per cent 
deviation from the theoretical values, are presented In Table 6: 

The theoretical approximations by the delta method agree fairly 

^ y ' ' ° r ' 

closely with the ei^plrlcal estimates of the standard etrors of r^.^^ , r^.^ » 

and . The largest differences occur in Instances ^ere negative sam- 
pie covariancea were more likely, that is, with p - .6 and N - 50. (Nega- 
tive aaaple covariancea wiat be assumed not to occur for valid application 
of the delta aethod.) The sane issue probably accounts for the empirical 
standard errors of. r being substantially greater than the theoretical 
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^ABLE 5 

Standard Errors of Monte Carlo Saapllng Distributions 

Congeneric 











p ■ .0 






■ .OjJ 










N - 50 


100 


' 200 


50 


100 


200 








.0960 


.0719 


.0515 


.0433 


.0307 


.0220 


K - 


A 




.0958 


.0717 ' 


.0515 


.0432 


.0307 


.0220 








.0972 


.0720 


.0515 


.0435 


.0308 


.0221 








0981 


.0730 


.0516 


.0435 


.0307 


.0220 






rx 
'f2 


, .0784 


.0613 


.0439 


.0380 


.0270 


.0184 


K » 


8 


.0770 


.0613 


.0438 


.0379 


.0270 


.0184 

« 








«0952 


.0688 


.0497 


.0434 


• OjU/ 


nono 
.UzU9 






• 0845 


.0640 


.0448 


.0386 


.0273 


.0185 


( 








Essential Tau-Equlvalence 










'fi 


.^842 


.0639 


.0464 


.0405 


* 

.0287 


'»' ' 

.0203 






'f2 


.0830 


'.0630 


.0462 


.0A04 


.0286 


.0203 


K - 


4 




.1133 


.0919 


.0648 


.0497 


.0349 


.0245. 






'a 


.0893 


.Q671 


.0476 


.0422 


.0292 


.0206 








.0845 


.0641 

♦ t 


.0465 


.0405 


.0287 


.0203 






'fi 


.0754 


.0594 


.0436 


.0375 


.026J 


.0184 


k - 


8 


'f2 


.0758 


.0581 


.0442 


.0372 


.0266 


.0184 






r - 
a 


.1133 


.0883 


.0646 


.0498 


.0344 


.0243 






.0775 


.0610 


.0443 


.0387 


.0271 


\ .0185 








.0816 


.0607 ' 


.0438 


.0377- 


.0267 


.0184 



TABLE 6 

/ 

Differences Between Monte Carlo and Theoretical 
Standard Errors, and Per. Cent Deviation 
from Theoretical Values * 



N 



— e — 
50 



Congeneric Measures 



100 



^Fl 



K - 8 'F2 



200 



.0038(3.8%) .0017(2.4%)' .0020(4.0%) 

-.0041(4.1%) .0014(2.0%) .0019(3.8%) 

.0036(3.5%) ' .0011(1.6%) .0015(3.0%) 

-.0095(10.8%) -.0005(0.8;?) .0003(0.7%) 

-.Oi09(12.4%) -.0005(0.8%) .0002(0,5%) 

-.0056(5.6%) -.0021(3.0%) .0003(0.6%) 



p = .833 



50 



.0013(3.1%) 
.0012(2.9%) 
.0012(2.8%) 

.0013(3.5%) 
.0012(3.3%) 
.0014(3.3%) 



100 



.0012(4.1%) 
.0011(3.7%) 
.0piO<3.4%) 

.0012(4.7%) 
.0012(4.7%) 
.0012(4.1%) 



200 



.0012(5.8%) 
.0011(5^3%)' 
. 0010(4. 8Z) 

.0002(1.1%) 
.0002(1 .It) 
.'0001(0.°5%) 



fau-Equivalent Measures 



K « 4 



K - 8 



r_, -.0088(9.5%) 

rp2. -.0100(10.8%) 

.0024(2.2%) 

r -.0037(4.0%) 
a 



-.0111(12.8%) 
-^.0107(12.4%) 
rj^ .0024(2.2%) 
-.0090(10.4%) 



.0016(2.4%) 


.0002(1.6%) 


.0013(3.3%) 


.0011(4.0%) 


.0008(4.1%) 


.0025(3.8%) 


.0000(0.0%) 


.0012(3.1%) 


.0010(3.6%} 


.0008 (4. It) 


. 0139(17. fi%) 


.0098(17.8%) 


.0027(5.7%) 


'.0019(5.7%) 


.0012(5.23:) 


.0017(2^4%) 


.0014(3.0%) 


.0030(7.7%) 


.0017(6.2%)' 


.0012(6.2%) 


.0014(2.3%) 


.'000.7(1.6%) 


.0013(3.6%) 


.0011(4.3%) 


.0004 (2. 2X) 


.0027(4.4%) 


.0013(3.0%) 


.0010(2.8%) 


.0010(3.9%) 


.0004(2.2%) 


.0103(13.2%) 


.0096(17.5%) 


.0028(6.0%) 


.0014(4.2%) 


.0010(4.3%) 


.0002(0,. 3%) 


.0014' (3. 37.) 


.0024(6.6%) 


. 0015(5. 9^i) * 


.0005 (2. a%) 



* Computed from data taken to six decimal places 
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value. With p « .833^ the cUa^epancy between empirical and theoretical 
standard errors was 1ms than 6 per cent for all twelve combinations of 
N and K . Undoubtedly this discrepancy is largely accounted for by the 
limltationjKof the delta method. ' 



/ 



^ Cbnaluaiona and Reocnmendations j 

The theoretical and empirical standard errors lead to the following 
conclusions: 

1) For the congeneric case' and K - 4, the standard errors of r^^ , ' 
r , r and r ^exhibit no important differences with samples : 

- p r J 

50 or greater. But with K » 8, the standard error of r^ is ^ub- 
stantially larger than those of r^^^- , v^^ and r^^g . 

2) In almost all cases the standard errors of r^^^ and r^^ are smaller 

than those of the other coefficients. This finding al8<?^ holds true 

/ 

under tau-equi valence, in*whicU r was included among ^he comparison! 

a 

3) The coefficients which maintain the identity of the ^rts and are 

/ 

based on the full set of part-test variances and co^ariances (r^^ , 

r . r • r,^) exhibit decreasing standard errors/ as the number 
F2 • a \ LS' y 

of part-tests increased from four to eight. The /standard error of a 

/ ' 

Kristof coefficient did not depend on the origlxial number of parts 
before combination. ^ 

4) In every comparison the standard errors of r^ii^^ and r^^ show no 
material difference. To achieve this approajlmate equality it may 
be necessary td use tb<». largest part-test t|o define the pivot row 
for tpj. 

i 
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The bias of all the coefficients vas also examined in the Monte 
Carlo study. None of the' coefficients exhibited any substantial bias 

except cocificient alpha under the congeneric case. The negative bias 

> 

of r in this situation was to be expected* 
a 

Which of these coefficients is to be preferred for a measure with 
congeneric parts depends on the factors of most importance to the 
reaearchor. In terms of computational simplicity, r^^ * 
distinct advantage over r^^ and r^^g • (Coefficient r^^g is probably 
Impractical without a computer routine for its computation.) In terms 
of their standard errors r^.^^ , r^^ and r^^g are preferable to rj^ • If 
uniqueness is considered an advantage—in the sense that for a given 
set Df data arbitrary decisions do not affect the computed reliability 
coefficient— then t^^ and r^^g are preferred ovar r^^ * 

As a compromise the authors favor r^^, ' ^° disadvantage is associ- 
ated wit*' this coefficient in terms of standard error. Computation is 
not difficult once the part-test variance covar lance matrix is available. 
Monte Carlo simulation data suggest that the row with the largest sum of 
elements serves well as the pivot row. If the researcher is uncertain 
whether the parts can be assumed to be tau-equivalent or can be regarded 

t 

* r 

to be only congeneric, r^^ ady^ntage that coefficient alpha 

might be thought -t^^'have. Coefficient ^° adequate as alpha for 
truly tau-«0'.ivalent 4>arts and quite superior to alpha with congeneric 
parts. 



42 



39 

REFERENCES 

Brovn^, Michael W. A comparison of factor analytic techniques. 

Psychometrtka , 1968, 33, 267-334. 
Cronbach, L.J. Coefficient alpha and the internal structure of tests. 

Psychoisetrika , 1951, 16, 297-334. 
Feldt, L.S. Estimation of the reliability of a test divided into two 

parts of unequal length. Psychometrika , 1975, 40, 557-561. 
Fisher, R.A. Moments an# product*^ moments of sampling distributions. 

Proceedings of the London Mathematical Society, 1928, 30, 199-238. 
Gilmer, Jerry S. Reliability estimation of a te6t with parts of unequal 

length. Unpublished -doctoral dissertation. University of Iowa, 1981. 
Gilmer, Jerry S. and Feldt, Leonard S. Reliability estimation for a test 

with parts of unequal and unknown lengths. Unpublished manuscript, 

1981. ' \ 

Horst, Paul Estimating total test reliability from parts of unequal, 

length. Educational and Psychological Measurement , 1951, 11, 368- 
371. 

Kendall, M.G. and Stuart, A. The Advanced Theory of Statistic s; Vol. I, 
Distribution Theory , Ord Ed.). London: Charles Griffin and 
Company, i969> 

Kristof , W. Estimation of reliability and true score variance from a 

f 4 

split of a test into three arbitrary parts. Psychometrika , 1974, 
39, 491-499. 



40 

Mayekava^ S. and Haebara^ T. Estimation of the reliability of a test 
consisting of B»re than three congeneric parLs. Iowa Testing 
Programs Occasional Papers, Number 28 , e University of Iowa, 
Iowa Cltyy Iowa, 1980. 

Novick, M.R. and Lewis, C- Coefficient alpha and the reliability of 
composite measurements. Psychometrika , 1967, 32, 1-13. 

Odell, P.L. and Feiveson, A.H. A ntanerical procedure to generate a 
sample covar lance matrix. Journal of the American Statistical 
Association . 1966, 61, 199-203. 

A * 



■ 1 

\ 

\ 

\ 



\ 



J 



/ 



44 



